Add Queue.workerLifetimeJitter to stagger worker shutdowns#476
Conversation
Each worker picks a random offset in [0, workerLifetimeJitter] at startup and adds it to its effective lifetime. Prevents thundering-herd restarts when a fleet of workers (e.g. ECS/Kubernetes tasks) is spawned at the same instant and would otherwise all exit on the same tick. Defaults to 0 (no jitter), preserving existing behavior. Idea and original implementation (against a legacy CakePHP 2.x fork) by Rommel Penaflor (@xrompdev) in #475; ported to the modern processor.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #476 +/- ##
============================================
+ Coverage 77.22% 77.39% +0.17%
- Complexity 949 966 +17
============================================
Files 45 45
Lines 3196 3247 +51
============================================
+ Hits 2468 2513 +45
- Misses 728 734 +6 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Pull request overview
Adds an optional Queue.workerLifetimeJitter configuration to stagger worker shutdown times by adding a per-worker random offset to the effective worker lifetime / --max-runtime, reducing synchronized restarts in large fleets.
Changes:
- Add per-worker lifetime jitter computation in
Queue\Queue\Processorand apply it to the effective max runtime. - Add unit tests for jitter offset bounds/default behavior.
- Document the new configuration option and add it to the example config.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
src/Queue/Processor.php |
Computes a per-worker jitter offset and adds it to the effective max runtime; logs when jitter is chosen. |
tests/TestCase/Queue/ProcessorTest.php |
Adds unit tests for jitter offset default/bounds/negative handling. |
docs/sections/configuration.md |
Documents workerLifetimeJitter usage and intent (staggered shutdowns). |
config/app.example.php |
Adds workerLifetimeJitter to the example configuration. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Summary
Adds an optional
Queue.workerLifetimeJitterconfig (seconds). Each worker picks a random offset in[0, workerLifetimeJitter]at startup and adds it to its effectiveworkerLifetime/--max-runtime. When a fleet of workers is spawned at the same instant (ECS tasks, Kubernetes deployments, systemd unit with many instances), this prevents every worker from terminating on the same tick and producing a thundering-herd of simultaneous restarts.Defaults to
0, so behavior is unchanged unless the option is set.Credit
Idea and original implementation by Rommel Penaflor (@xrompdev) in #475, where it was proposed against a legacy CakePHP 2.x Symphosize fork and therefore could not be merged directly. This PR is a fresh port to the modern
Queue\Queue\Processor, keeping the operational intent intact.Implementation notes
$startTime = time()inProcessor::run()), not re-rolled each loop iteration, so the exit time is stable per worker.Processor::computeLifetimeJitterOffset()so the bounds/default behavior is unit-testable without spinning up the full run loop.$maxRuntime > 0— unlimited workers stay unlimited.<= 0jitter values are ignored (returns 0), so a misconfigured negative value is a no-op rather than an error.Applying worker lifetime jitter: +Ns secondsso operators can see the stagger in action.Docs
Added a dedicated bullet in
docs/sections/configuration.mddirectly after theworkerLifetimesection, explaining the ECS/K8s use case.